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ABSTRACT 

The Service Delivery Assessment (SDA) model is a 
human services evaluation tool which offers a viable alternative to 
core traditional approaches and is appropriate for small local 
evaluations as well as large national studies. There are five phases 
to an SDA study: assignment, pre-assessment, design, analysis, and 
communication of^ the findings. Assignments are decided by the 
Secretary of Health and Human Services and the Inspector General, and 
may request answers to highly specific questions or be quite 
open-onded. During the pre-assessment period, information is gathered 
aboac local conditions, and possible contacts, respondents, and site 
study settings are determined. Because the primary means of gathering 
information in an SDA study is the one-to-one personal discussion, 
the design phase focuses on identifying a wide array of people with 
whom to talk; in addition, SOA teams observe local conditions and 
activities during this period. The analysis phase is best described 
as one of consensus building by iterative steps; as information is 
gathered, trends and patterns emerge and new information is sought 
that will verify or repudiate investigators* hunches. The final 
report includes th^ee different types of information — a description 
of local activities and conditions, a comparison of conditions with 
standardized or existing critr^ria, and the interpretations of the SDA 
team. Six characteristics of the SDA method ^re provided, as well as 
suggestions on when to use this approach. Finally, the evaluation of 
a teacher training summer institute is described to illustrate the 
application of the SDA model. Two references are appended, (JB) 
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The critical feature of human services is that they arp 
highly opera tor -dependent and difficult to standardize. 
Hence it is always probleoiatic whether a treatment is 
being delivered as designed, whether the mode of the 
delivery is adding some unintended treatment to the 
basic one, and finally, whether the treatmenc can be 
delivered in a reasonable way at all • • • ' 

Rossi 



THB SERVICE DBLIVBRY ASSESSMENT MODEL 



When top management has to depend totally on 
abstractions, such as formal reports, figures, and 
quantitative data, rather than be able to see, know, 
and understand the busine88~its reality, its people, 
its environment, its customers, its technology — then a 
business has become too complex to be manageable* A 
business is manageable only if top management is 
capable of testing against concrete reality the 
measurements and information it receives* 

Peter Drucker 



Thi' idea of Service Delivery Assessment was born in 1967 when 
Joseph walifano was appointed by Lyndon Johnson to be head of the 
Department of Health, Education and Welfare (HEW) — the then 
largest cabinet department* Experienced in working with large 
government bureaucracies, Califano resolved not to be insulated 
from the actual conditions and daily activities of those in his 
department working at "the client's end of the service pipeline*" 
Accordingly, within the office of the Inspector General, he 
established a team of investigators to carry out what he called 
service delivery assessments (SDAs) * 



The SDA team was located in ten regional offices around the 
country, with a central office in Washington* Throughout the 
year, the fifty or so investigv^tors ware given specific 
assignments to carry out short-term, but intense, studies of 
programs in the areas of health care, income maintenance, 
education, and social services* These teams were given 
considerable freedom in the way they conducted their studies # and 
were not asked to replace traditional program evaluators or 
program monitors. Rather, they were asked to provide another 
perspective by closely examining local conditions, activities, 
and perspectives* 



There are five phases to an SOA study*— ass igrunent, 
pre^-assessment, design, analysis # and the communication of the 
findings. Each of these phases is discussed below. 



1, Assignment 

Working together, the Secretary of HEW (later changed to 
HealtA and Human Services (HHS)) and the Inspector General meet 
at the beginning of the year to outline the areas and topics of 
potential studies. Restricted patient admittance to nursing 
homes 7 low income energy assistance, medicare physician 
assignment, child abuse, and developmental disabilities are 
typical of the issues examined in SDA studies. In addition to 
the topics outlined at the start of the year, the service 
delivery assessment teams respond to those topics identified 
during the year by the Secretary. Por example, the Secretary may 
ask for information on a program that has sudden operational 
difficulties, or information on emergency situations (such as the 
arrival of the Indochinese refugees), which may necessitate a 
study on very short notice. 



The assi^ ime:it may request answers to highly specific 
questions (for ^ixample. What is the quality of the toll-free 
service for medicare beneficiaries?), or the assignment may be 
quite open-ended, requesting information about conditions in a 
Urge domain (for exanqple. What is the present state of civil 
rights enfo;. cement? ) . 



2 . Pre-Assessment 

One regional office is designated to serve as the project 
leader, and three or four other regional offices are assigned to 
support the project* The first task for the lead office is to 
conduct a pre-assessment study aimed at gathering an 
understanding of local conditions, and at generating a list of 
issues to pursue during the investigation. The service delivery 
assessment office spends considerable time and resources on this 
phase of the study, determining possible contacts, respondents, 
and site study settings. Site visits are scheduled so that SDA 
staff can "get their feet wet," and local program officials are 
visited so that their perspectives on present program operations 
can be gathered. Finally, a brief review of relevant documents 
and literature is carried out. 



In addition to checking out thn scene, the SDA lead team 
spends some time with those officials in the cabinet close to the 
Secretary, attempting to discover ^Jie views and interests of the 
policymakers who will ultimately use the findings. In this way, 
the SOA team works from both ends, so that there is no mismatch 
between the needs of the policymakers and information generated 
by the study. 



3« Designing the Study 



1 



Having studied the results of the pre'-assessment, the lead 
team designs the full study. The primary means of gathering 
information in an SDA study is one<-to-one personal discussions 
(these discussions are not always as formal as structured 
interviews). Gaining a wide and full variety of perspectives is 
a top priority, so that sometimes considerable imagination is 
used in selecting a wide array of people to talk v/ith. In one 
study of domestic violence, the SDA teams talked with hospitals, 
shelters for battered women, private homes, police stations, 
jails, judges, community centers, city service agencies, schools, 
township government oftices, and crisis intervention teams. 



In addition to talking to a wide range of people, SDA teams 
observe local conditions and activities. In a study of migrant 
farm workers, for example, much was learned by observing the 
living conditions and the interactions between the workers and 
local health care officials. In observations of local school 
board meetings, one SDA team member discovered how allocations 
were and were not being made for handicapped children. 

Examination of documents can also be illuminating, showing, 
for example, the complexity of forms and procedures local welfare 
recipients must endure. One examination of local program memos ) 
showed how the communications to the head office were 
systematically biased in their reports of services rendered. 

In choosing which sites to visit, or which people to talk to, 
random sampling may not be used. Rather, 3ites and participants 
are selected on the basis of the overall study needs. For 
example, in the investigation of extreme program practices, 
atypical sites may be specifically sought out. Both judgment and 
selective sampling are used to ensure that a full lange of 
program practices and participant perspectives are included. 

Also, the SDA study has the freedom to follow whatever leads 
they encounter in the field. They operate very much in the mode 
of the investigative journalist (see Guide No. 6 on investigative 
journalism). To capitalize on their freedom to explore all 
leads, the SDA regional site teams convene about one-third of the 
way through the study to discuss what they have seen, and to 
share interesting leads. In this way, the teams have both new 
and shared issues to 'westigate when they begin the second round 
of site visits and i<.cerview8. 
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4> Analysis and Synthesis 



Unlike the handling o£ statistical data in experimental 
studies f there are no set algorithms for handling the data 
gathered by the SDA teams. The process of qualitative analysis 
is best described as one of consensus building by iterative 
steps . As information is gathered, trends and patterns emerge. 
New information is sought that will verify or repudiate 
investigators* bunches. Again, the parallel with the methods of 
investigative journalism is most pronounced in this respect. 

Consensus is arrived at through several means. First, each 
regional team %#orks up its own report independent of the other 
teams. Second, the raw field data (in the form of transcribed 
interviews or coded responses) are forwarded to the lend team 
office. Third, and most important, all key regional personnel 
meet in a two- to three-day debriefing meeting where all the 
issues are discussed, the key findings summarized, and a report 
outlined. 

In a service delivery assessment (SDA) study, then, analysis 
and synthesis is an ongoing process — it begins with the 
individual investigators, moves to the regional offices, and 
finally takes place on a national level. Throughout the study, 
this process is interactive, iterativ<i, and accomplished largely 
through sharing and discussion of personal perceptions. 

Communicating the Findings 

The first means by which the findings are communicated to the 
Secretary is a short (10-20 pages) written report. A draft of 
the report is circulated among the Secretary's other cabinet 
runagers, so that their recommendations can be incorporated 
before the report goes to the Secretary. 



In general, there are three different kinds of information 
included in the final report. First, the report provides a 
description of Tocal conditions and activities. Here the SDA 
simply serves as the eyes and ears of the Secretary. Second, the 
report attempts to put these findings in perspective by comparing 
conditions with some met of hypothesized standards, with the 
original aims or goals of the program, with other similar 
programs, or with past levels of service. Third, the SDA team 
tries to interpret what it sees. Its interpretations might 
include guesses about causes and effectr;, assessments of strengths 
and weaknesses, and recommendations for changes. These interpre- 
tations are kept quite separate from the other findings, and the 
Secretary is, of course, free to accept them or not. 



The second means of communication is a one-hour briefing 
scheduled with the Secretary. With the aid of charts and 
nandouts, the essence of the findings are communicated in a 
20-iLiAUte presentation. Approximately 40 minutes are then left 
for questions and answers. There is usually a practice briefing 
with the other department managers prior to the briefing for the 
Secretary. 



CBARACTBRISTICS OF THE SDA METHOD 

A service delivery assessment (SDA) study is characterized by 
the following features: 

• is typically designed to produce answers to 
policymakers* questions in a short time (a month or two), 

• serves as the eyes and ears of program sponsors and 
policymakers, 

• relies primarily on a flexible mixture of interviews, 
observati<ni8, and document analyses to gain "snapshots* 
of daily realities, 

• is open-ended and gently jnvestigctive in nature, 
following up on issues that emerge, 

• relies on briefing and debriefing as the main means of 
analyzing and synthesizing information, 

• is most often used to supplement other means of 
gathering evaluative information. 



NHBM TO USB TBB SDA APPROACH 



Use the SDA approach when 

• progreun decision- makers are at a distance from the daily 
operations of the program (for example, when the program 
is very large) ; 

• policymakers need answers to specific questions about a 
program; 

• a program is in a trial period or in its early stages 
and changes can be made; 

• an evaluation study is to be primarily in-house and the 
aim is to improve one*s own operations; 



• a program has unique qualities or is particularly 
successful (or unsuccessful) and a full understanding or 
analysis is desired; 

• emotional or political issues emerge and a deeper f 
almost investigative, study of the program is desired; 

• other, more formal methods distort or leave a gap in the 
understanding of the essential character of the program. 



SFRVICB DBLIVBRY ASSBSSNBRT FOR TCB EVALUATOR 



The SDA model is useful for the evaluator, offering a viable 
alternative to more traditional approaches. Its methods are 
flexible, adhering more to rules of common sense than research 
protocol. The model is appropriate for small local evaluations 
as well as large national studies. The essential ingredients of 
the model are outlined in Table 1. 



In addition to its usefulness as a general approach, the SDA 
model suggests some specific techniques that can be adopted and 
adapted by the evaluator of educational programs. One of these, 
the oral briefing, is particularly valuable, and is discussed 
more fully in Table 2. 



AN EXAMPLE 



The following example is given to illustrate the utility and 
flexibility of the basic approach contained in the SOA model. As 
long as the key characteristics of the model (the investigative 
nature, the use of qualitative methods, the entasis on oral 
briefings) are retained, then the SDA model can be freely and 
imaginatively adapted to fit any local situation. The model is 
an inherently sensible one and is very appealing to local 
personnel. 



It was decided to use SDA methods in the evaluation of a 
teacher trainin. sumer institute to be held in a large and 
prestigious science museum. Sponsored by the National Science 
Foundation (NSF) , the goals of the training program were to ah ^ 
teachers how to use the museum as a resource for their high 
school classes, to teach the teachers some new physics concepts, 
and to reward them with an enriching and inspiring experience. 
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Table 1 

The Ser/ice Delivery Assessment Model 
Five Phases of a Stxidy 



1. Assignment 

(a) Topics are determined by: 

• policymaker's request 

• emergency needs 

• field discoveries 

(b) Research team is assiqned 
questions to study like: 

• what are conditions and 
activities like? 

• what are barriers "^o 
effective service? 

• ii^at are the best 
iworst) practices? 

• what are the perspec- 
tives of these involved? 

2 . Pr e*"Assessmen t 

(a) A quick field study aims to: 

• understand functioning of 
the program 

• understand local conditions 

• conduct short literature/ 
document review 

• initially frame issues and 
questions for study 

(b) Policymakers* needs are 
assessed: 

• assignment review 

• discussion with close staff 

3 . Design 

(a) A meeting is held to plan: 

• choice of sites 

• choice of participan' 

• summary techniques px med 

• consensus meetings 
scheduled 



(b) Site visits are made to 
carry out: 

• discussions 

• observations 

• review o£ documents 

• pursuing of leads 

4 . Analysis 

(a) The research team seeks 
consensus by debriefing: 

• at the individual 
level 

• ac the team level 

• at the project level 

• as an ongoing 
iterative process 

(b) independent reports are 
written at the team level 

(c) All field data is 
aggregated and analyzed 
at project level 

5* Communication of Findings 

(a) A report is written that 
is: 

• submitted within 90 
days of assignment 

• short with essentials 
summar ized 

(b) Policymakers receive an 
oral policy briefing # 
including: 

• description of condi- 
tions and activities 

• compar isons 

• interpretations 
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Table 2 
Notes on Oral Briefings 



1. Se3 *ctinq and Orgauizing 
Infurmation 

Information should be pertinent 
to the needs of the 
policymakers. 

Only what is key to under- 
standing the program should be 
presented. 

Only infornation that is 
related to feasible courses of 
action should be presented. 

2. Selecting the Presenter 

The person should be an 
effective speaker, with indepth 
knowledge of the study. 

The persons should have confi*- 
dence# poise;, and the ability 
to handle the unexpected and 
inevitable distractions. 

3. Preparing the Presentation 

If possible, the presenter 
mould give practice briefings 
to the staff of the policy- 
maker and incorporate the 
questions and concerns they 
have. 

Charts illustrating the key 
findings should be prepared. 

Handouts identical to the charts 
will aid the participants in 
following th^ briefing, as well 
as leave them with a written 
summary of the briefing. 

Slides, videos, or other media 
can be very effective in 
communicating the reality of 
the program* s main issues. 



4. Setting the Stage 

A short report with an 
executive summary should be 
sent !:o participants just 
before the briefing. 

An agenda for the briefing will 
establisn time periods for the 
different phases of the 
briefing. 

5. Selecting Participants 
Participants should include: 

• the chief policymaker (s) 

• other managers 

• key program staff 

• a figure who can naturally 
serve as a liaison between 
the evaluators and the 
policymakers 

• other key figures 
representing special 
interests 

6. Holding The Briefing 

The briefing should proceed as 
follows: 

• a short presentation summa- 
rizing highlights, findings, 
compar isons , interpretations 

• a short time for program 
staff to respond 

• a longer period for 
questions, answers, and 
discussion 

7. Recording and Followup 

The meeting should be recorded 
and notes taken. 

Particular attention should be 
given to suggestions that call 
for followup actions. 



ERLC 
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The course was administered by the museum and taught by two 

university physics instructors. The program was funded for a ) 
three-year period and was to provide in-service assiscance to the 
teachers during the year, as well as conduct the summer workshops. 
In the first summer, two different four-week workshop sessions 
were held. 

NSF had asked that an evaluation be done, but the assignment 
to the evaluator was vague. As no one was quite :«ure what 
exactly would happen in the workshops, the director wanted the 
evaluator to "keep an eye on things" and eventually "help the 
teachers find their way." It was felt that it was too early for 
any kind of summative report to be made to NSF, so that it made 
more sense for the evaluator to serve as an in-house consultant 
and to carry out a service delivery assessment (SDA) study. 

The charge to the evaluator was, at least for the first year, 
to provide feedback to the museum program officials, and to the 
teachers, so that the program would get off to a good start. It 
was also hoped that the evaluator would be able to make recom- 
mendations about ideas for followup and inservice activities. 

In the pre-assessment phase, before the start of the program, 
the evaluator talked with the t'3achers and the program director 
in order to learn what they thought the program would or should 
be doing. Documents, including the initial progreun proposal, the 
applications of the participants, and a sample of the teaching 

materials, were all reviewed. In the first week of the workshop, ' 
the evaluator attended as many of the sessions as possible, 
acting as a participant. During this period, the following 
issues emerged from the evaluator *s observations. 

• The participants, supposedly experienced physics 
teachers, were in fact a widely diverse group— some 
having little or no experience in physics and math. 

• The course, irtiile ostensibly aiming to teach 
methods of exhibit-based discovery learning, 
consisted largely of almost standard physics 
lectures. 

• Because of logistics and econ<MBics, many of the 
teachers would not be able to bring their students 
to the museum regularly during the school year. 

The evaluator discussed these observations with other members 
of the museum staff who were sitting in on the workshop, and 
confirmed many of the observations. The conclusions were 
presented to the associate director of the museum, who encouraged 
the evaluator to begin giving feedback to the university 
instructors. A meeting was held with the instructors, and in a 
week or so the workshop had evolved into a more interactive and 
exhibit-oriented approach. 
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After the first week, the evaluator began to talk with the 
participants about the mrkshop, and how it was going for them. 
After several weeks, the evaluator set with the director, and 
discussed what had been seen and heard. It was the director's 
opinion that the evaluator should continue with the interviews, 
trying to talk with everyone in the institute. At the sane time, 
the director was particularly eager to get ideas for in-service 
activities. In the meantime, the evaluator was to continue 
providing feedback to the instructors. 

The interviews and observations continued, with feedback 
given to the instructors each week. Toward the end of the 
course, the instructors were given ticanscripts of all the 
interviews with the participants. This provided them with full 
access to the perceptions of the participants, and w ; for them 
an unusual and very rich source of feedback. The instructors 
were asked to read through the transcripts and to respond to them 
at the final briefing meeting, which was to be held shortly after 
the conclusion of the workshops. 

Present at the briefing were the director, associate director 
(who served as liaison between the evaluator and program staff) , 
participating staff, and instructors of the institute. For 
approximately one-half hour the evaluator presented a summary of 
the findings. The course was described from the evaluator *s 
perspective; the »«orkshop was compared with other similar NSF 
summer workshopsi and interpretations were offered of what had 
been the strengths and weaknesses of the workshop. For example, 
the evaluator made the following points: 

e While the course addressed participants of all 
levels well, the workshop was more successful for 
those more advanced in physics. 

e There appeared to be a double-bind, in that some 
teachers wanted more specific teaching ideas for 
their classess, and less time spent on learning 
physics. However, they agreed that such ideas were 
not worth much without a thorough understanding of 
the physics behind them. Thus, it appeared to be 
difficult to do both in the short time of the 
institute. 

The evaluator also made suggestions, ranging from the major 
to the trivial, for next summer's iforkshops. Finally, recom- 
mendations were made for in-service activities. The meeting was 
not without interruption or diversions, but the main points were 
covered and discussed. 
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I 

How effeccive was the SUk approach ana the oral briefing? in 
a very small evaluation of tiMs kind, it offered the advantages ) 
of letting the evaluator Jocus time and energy where it could be 
of most use to the program. Both the directors and the instruc- 
tors felt well informed about the progress of the course, even to 
the extent of having issues raised which they would rather not 
have to face. In this way, the evaluator served as a kind of 
catalyst, making sure that problems or difficulties were not 
ignored. Moreover, both the directors and the instructors used 
the evaluator to get answers to specific questions they had about 
how the participants were doing and learning. The director asked 
the ^valuator to write a short descriptive report documenting the 
flavor of the first summer's workshops. Both evaluator and staff 
f^lt thit, contrary to their experience with other evaluations, 
this had been an honest and useful effort. 
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